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TITLE OF THE INVENTION 



IMAGE SYNTHESIS METHOD, IMAGE SYNTHESIS APPARATUS, 
AND STORAGE MEDIUM 

BACKGROUND OF THE INVENTION 

Field of the Invention 

The present invention relates to technology for 
combining a plurality of images in which adjacent images 
have a common subject region and generating a synthesized 
image . 

Description of the Related Art 

Hitherto, the synthesis of a panoramic image by 
combining a plurality of images is performed by a 
conventional method of capturing images of a subject in a 
plurality of regions and synthesizing a single image using 
image information about a common subject region, as shown in 
Fig. 10. Fig. 11 shows a sectional view taken through the 
XZ plane of Fig. 10. As shown in Fig. 11, an image 
synthesis method includes a mode (planar mapping mode) of 
mapping captured images onto a single plane 800 and 
synthesizing an image or a mode (cylinder mapping mode) of 
mapping the images onto a virtual cylindrical surface 801 
having a radius f and synthesizing an image. Fig. 12 



schematically shows a synthesized image obtained when the 
images are mapped onto the plane 800. 

In the conventional panoramic image synthesis method, 
it is necessary to synthesize an image in a predetermined 
5 mapping mode. Alternatively, a user is required to manually 

set a mapping mode. 

For example, when a plurality of images captured by 
horizontal panning are combined into a synthesized image in 
the planar mapping mode, as shown in Fig. 12, an image with 

10 distorted edges is generated due to a wide horizontal 

viewing angle. When a synthesized image is formed on a 
region in which the horizontal viewing angle is 180° or 
greater in the planar mapping mode, the synthesized image 
cannot be mapped on a single plane, possibly causing a 

15 serious apparatus failure. 

It is burdensome to conform to various imaging modes, 
i.e. , a horizontally-rotating (panning) mode shown in Fig. 
10, a vertically-rotating (tilting) mode, and a 
horizontally/vertically-rotating (panning/tilting) mode, and 

20 to designate an appropriate mapping mode from among these 

mapping modes . When the user is not knowledgeable about 
these mapping modes, the user may fail to select the 
appropriate mapping mode. 

In the conventional panoramic image synthesis method, 

25 an appropriate mapping method is selected and set prior to 
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generating necessary parameters for synthesis processing. 
Using this mapping method, an input image is mapped and 
transformed. Subsequently, the parameters indicating the 
positional relationship among a plurality of images are 
5 generated based on corresponding point information, and a 

;M synthesized image is generated. 

When the mapping method is subsequently changed and a 
j= synthesized image is again generated since the previously- 

=n selected mapping mode is inappropriate, it is necessary to 

10 re-input the images, to perform mapping transformation in 

f 1 

"1 another mapping mode, and to perform complicated image 

synthesis processing. In other words, it is necessary to 

::f again perform processing from the beginning to generate the 

parameters . This places a heavy computational burden on the 

15 apparatus, requires a long processing time, and is extremely 

inefficient . 



SUMMARY OF THE INVENTION 



20 Accordingly, it is an object of the present invention 

to solve at least one of the above problems. 

Another object of the present invention is to provide 
an image synthesis method, an image synthesis apparatus, and 
a storage medium in which a mapping mode is easily set when 

25 synthesizing an image. 
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Another object is to enable a user to select a mapping 
mode with a simple operation so as to select an appropriate 
mapping mode without detailed knowledge. 

According to an aspect of the present invention, the 
5 foregoing objects are achieved through provision of an image 

synthesis method including a placement information obtaining 
step of obtaining placement information about a plurality of 
images in which adjacent images have a common subject region, 
and a synthesis step of combining the images using a mapping 
10 mode in accordance with the placement information. 

In order to perform more accurate image synthesis, the 
image synthesis method may include a focal length obtaining 
step of obtaining focal length information of each of the 
images. In the synthesis step, the mapping mode may be used 
15 in accordance with the focal length information and the 

placement information. 

Another object of the present invention is to change 
the mapping mode to an appropriate mapping mode even when 
the mapping mode is not appropriately selected using the 
20 placement information or when the selected mapping mode 

differs from a mapping mode preferred by the user. 

To this end, the image synthesis method may include a 
changing step of changing the mapping mode. 

It is another object of the present invention to 
25 inhibit the user from performing inappropriate image 



synthesis when the mapping mode is inappropriately changed . 

To this end, the image synthesis method may include a 
generating step of issuing, when an image formed by changing 
the mapping mode in the changing step does not comply with a 
5 predetermined condition set in accordance with the mapping 

mode, a warning and generating a synthesized image in 
accordance with the predetermined condition. 

Another object of the present invention to provide a 
visually beautiful image after synthesizing an image. 

10 To this end, the image synthesis method may include a 

displaying step of displaying a cuttable rectangular region 
without a margin when the synthesized image is displayed. 

It is another object of the present invention to 
provide an image synthesis method, an image synthesis 

15 apparatus, and a storage medium in which the mapping mode is 

quickly changed. 

According to an aspect of the present invention, the 
foregoing objects are achieved through provision of an image 
synthesis method including a generating step of generating, 

20 prior to performing mapping transformation of input images, 

coordinate- space transformation parameters for transforming 
a coordinate space of one image among the images into a 
coordinate space of another image, and an image synthesis 
step of combining the images based on a given mapping mode 

25 and the coordinate -space transformation parameters. 
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It is an object of the present invention to change the 
mapping mode to an appropriate mapping mode even when the 
mapping mode is not appropriately selected or when the 
selected mapping mode differs from a mapping mode preferred 
5 by the user. 

To this end, the image synthesis method may include a 
changing step of changing the mapping mode. In the image 
_?= synthesis step, coordinate transformation parameters set for 

each mapping mode may be changed and the input images may be 
ID again combined in accordance with a mapping mode changing 
instruction. 

Another object of the present invention is to enable 
the user to arbitrarily select a reference position of a 
synthesized image. 
15 To this end, the image synthesis method may include a 

reference position setting step of arbitrarily setting a 
reference position of a synthesized image. In the image 
synthesis step, the images may be combined in accordance 
with the reference position set in the reference position 
20 setting step. 

Another object of the present invention is to smoothly 
restore the mapping mode to the previous mapping mode after 
the mapping mode is changed. 

To this end, the image synthesis method may include a 
25 storage step of storing a generated panoramic synthesized 
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image, and coordinate transformation parameters and the 
coordinate- space transformation parameters, which are used 
for generating the panoramic synthesized image, every time 
the synthesized image is generated in the image synthesis 
5 step. 

:;, It is another object of the present invention to 

|f= smoothly restore the synthesized image to the previous 

^ condition after processing the synthesized image. 

f: : ! According to another aspect of the present invention, 

lb the foregoing objects are achieved through provision of an 

ri image synthesis apparatus . The image synthesis apparatus 

V" : combines a plurality of images into a synthesized image. 

2 Every time the synthesized image is edited, a newly 

generated synthesized image is stored. 
15 As described above, an appropriate mapping mode can be 

easily set. A panoramic synthesis image is obtained without 
special knowledge. The mapping mode can be selected in a 
more accurate manner. A user can intentionally select a 
different mapping mode. Hence, flexibility is provided in 
20 selecting the mapping mode. When the user inappropriately 

changes the mapping mode, the user is warned of the 
incorrect operation and obtains the synthesized image. It 
is possible to cut a region without a margin, and that 
region can be printed or stored. A synthesized image can be 
25 generated in any mapping mode by using common coordinate- 
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space transformation parameters and performing processing in 
accordance with the mapping mode. Because computation of 
the coordinate- space transformation parameters is omitted 
when the mapping mode is changed, the processing time is 
5 substantially decreased when the mapping mode is changed. 

It is possible to generate a synthesized image of enhanced 
quality because the synthesized image need not be 
transformed again. The user is enabled to set a reference 
position at a desired position when generating a synthesized 

"10 image. This flexibly implements geometric correction, such 

as tilting and shifting correction. By reading a number of 
synthesized images and coordinate -space transformation 
parameters generated by changing the mapping mode or setting 
the reference position, a synthesis result can be obtained 

15 at an arbitrary point. The previously- generated images are 

quickly read, thereby improving efficiency in the editing 
operation. 

Further objects, features and advantages of the present 
invention will become apparent from the following 
20 description of the preferred embodiments with reference to 

the attached drawings . 

BRIEF DESCRIPTION OF THE DRAWINGS 

25 Fig. 1 is a block diagram of a panoramic image 
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synthesis method and apparatus according to a first 
embodiment of the present invention; 

Fig. 2 is a schematic illustration of a placement 
screen; 

5 Figs. 3A to 3C are schematic illustrations of placement 

;i patterns in mapping modes ; 

T| Fig. 4 is an illustration of synthesis parameters; 

j= Fig. 5 is a flowchart showing a process performed by an 

}]] image synthesis processor of the first embodiment; 

lb Figs. 6A to 6C are illustrations of synthesized images 

ir1 obtained by mapping; 

Fig. 7 is an illustration of vertical- cylinder mapping; 
li Fig. 8 is an illustration of spherical mapping; 

Fig. 9 is a block diagram of a panoramic image 
15 synthesis method and apparatus according to a second 

embodiment of the present invention; 

Fig. 10 is a schematic illustration of conventional 
image capturing ; 

Fig. 11 is a schematic illustration of conventional 
20 mapping synthesis; 

Fig. 12 is a schematic illustration of a conventional 
planar mapping synthesis mode; 

Figs. 13A and 13B are schematic illustrations of 
synthesized images obtained by vertical- cylinder mapping; 
25 Fig. 14 is a schematic illustration of changing the 
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image synthesized by the vertical-cylinder mapping to a 
planar mapping image; 

Fig. 15 is a schematic illustration of a synthesized 
image within a displayable range; 
5 Fig. 16 is a schematic illustration of image 

information displayed for setting the displayable range; 
Fig. 17 is a block diagram of a panoramic image 
; synthesis method and apparatus according to a third 

embodiment of the present invention; 
10 Fig. 18 is a block diagram of a panoramic image 

synthesis method and apparatus according to a fourth 
embodiment of the present invention; 

Fig. 19 is a schematic illustration of panoramic image 
capturing; 

15 Fig. 20 is a schematic illustration of mapping 

transformation ; 

Figs. 21A to 21C are illustrations of mapping modes; 
Fig. 22 is a flowchart showing a process for generating 
synthesis parameters; 
20 Fig. 23 is a schematic illustration of synthesis 

transformation processing; 

Figs. 24A and 24B are illustrations of image synthesis; 
Figs. 25A and 25B are illustrations of image 
transformation in which a reference position is changed; 
25 Figs. 26A and 26B are illustrations of the image 
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transformation in which the reference position is changed; 

Figs. 27A and 27B are schematic illustrations of 
captured images in which an adjoining part is corrected; 
Figs. 28A to 28C are schematic illustrations of 
5 correction processing; 

o Fig. 29 is a schematic illustration of a synthesized 

tpi image obtained by changing/correction processing; and 

.?= Fig. 30 is a schematic illustration of the 

^ changing/correction processing sequence, 

lb 

DESCRIPTION OF THE PREFERRED EMBODIMENTS 

li Fig. 1 schematically shows a panoramic image synthesis 

method and apparatus according to a first embodiment of the 

15 present invention. Referring to Fig. 1, an image input unit 

101 inputs a plurality of images to be used for image 
synthesis. A placement information setting unit 102 sets 
the placement relationship among the input images . A 
mapping mode setting unit 103 automatically sets a mapping 

20 mode for synthesizing an image based on the information from 

the placement information setting unit 102. An image 
synthesis processor 104 generates synthesis parameters among 
the images and generates a synthesized image. A synthesized 
image display unit 105 displays the synthesized image formed 

25 by combining the images based on the synthesis parameters. 
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A synthesized image storage unit 107 stores the resultant 

synthesized image in a recording medium 121 or the like. 

The image input unit 101 inputs a plurality of images. 

The images can be input by reading images recorded in a 
5 recording medium, such as a magnetic disk. Alternatively, 

q the images can be read from a non-volatile memory or the 

=k like that maintains image signals obtained through an image 

l : capturing system, such as a digital camera. The read images 

:!t are temporarily stored in a memory 120. The images are read 

10 from the memory 120, and they are displayed on a display 122 

;i for displaying the placement of the images. Fig. 2 

; J schematically shows a placement screen 200. A user places 

=-f the images on the screen 200. The placement can be done by 

moving the images on the screen 200 using a device, such as 
15 a mouse 123. Also, it is possible to input a code for 

indicating the position of each image for performing the 

placement . 

A camera or the like can record information about the 
placement at the same time as capturing the images. The 
20 placement position recorded with the images can be read when 
the images are input. Based on this placement position, the 
placement information can be automatically set. 

Figs. 3A to 3C show placement patterns. Fig. 3A shows 
a placement pattern formed when the images are captured in 
25 the one -dimensional horizontal direction. Fig. 3B shows a 
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placement formed when the images are captured in the one- 
dimensional vertical direction. Fig. 3C shows a placement 
pattern formed when the images are captured in a two- 
dimensional tiled arrangement. 
5 The mapping mode setting unit 103 sets a mapping mode 

f ., for synthesizing a panoramic image based on the placement 

;=f information set by the placement information setting unit 

102. 

J;f Figs . 3A to 3C illustrate an example of setting a 

lb mapping mode according to the placement of the images . The 

& mapping mode is set as follows. When the placement of the 

riJ images is in the one-dimensional horizontal direction, as 

O shown in Fig. 3A, the images are mapped as if they are 

pasted onto a surface of a vertical cylinder. When the 
15 placement is in the one-dimensional vertical direction, as 

shown in Fig. 3B, a synthesized image is mapped as if it is 
pasted on a surface of a horizontal cylinder, which is a 
horizontally-aligned cylinder. When the images are arranged 
in the two-dimensional tiled placement, as shown in Fig. 3C, 
20 the images are mapped as if they are pasted on the surface 

of a sphere shown in Fig. 3C. 

Accordingly, when the images are obtained in the one- 
dimensional horizontal direction, a synthesized image will 
have a wider horizontal viewing angle (360° maximum). Hence, 
25 the vertical-cylinder mapping is set, as shown in Fig. 3A, 
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and the synthesized image can display the entire image 
region- Similarly, when the images are provided in a 
different placement pattern, the synthesized image can 
display the entire image region by automatically setting an 
5 appropriate mapping mode according to the placement pattern. 

After the mapping mode is set by the mapping mode 
setting unit 103, as described above, the image synthesis 
processor 104 performs image synthesis processing. 

=: Subsequent to the confirmation of the mapping mode, 

10 synthesis parameters are generated by a method as described 
in Japanese Laid-Open Patent No. 9322040. Fig. 5 shows a 
flowchart showing a process performed by the image synthesis 
= processor 104. 

In step S501, corresponding points are extracted using 

15 image information about a subject region shared by two 

adjacent images as shown in Fig. 4. The corresponding 
points can be extracted by a known technique, such as a 
correlation method or a template matching method. 

The synthesis parameters are generated using the 

20 corresponding point information obtained in step S501. The 

parameters can be generated as follows. As shown in Fig. 4, 
two adjacent images 401 and 402 have a point P (x, y, z) on 
the image 401 and a corresponding point P' (x' , y' , z*) on 
the image 402, respectively. The relationship between 

25 pixels on the two images 401 and 402 can be expressed by 
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eight parameters (ml to m8) as follows: 



m-^ m 2 m 3 
m 4 m 5 m 6 
m 7 m a 1 , 



y 



(1) 



Eight simultaneous equations can be solved by using the 
equation (1) and the coordinates of the corresponding points, 
thereby computing the eight parameters (ml to m8). 
Alternatively, the parameters can be estimated more 
accurately by performing optimization, such as a least- 
squares method, using information about numerous 
corresponding points. 

When the point P' is mapped on a plane at the distance 
of one, which is the same distance as the image plane on 
which the point P is disposed, the X-Y coordinates of (u' , 
v" ) are obtained as u' = x'/z' and v' = y'/z'. 

The parameters generated in step S502 give relational 
expressions between the two images. Specifically, the 
relational expressions are obtained as parameters based on 
the coordinate system of the image 401 in Fig. 4. In step 
S503, these parameters are converted into parameters (Ml to 
M8) with respect to the reference coordinate system used 
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when combining all the images . 

In steps S504 and S505, the process is branched to 
three mapping synthesis processing steps in steps S506 to 
S508 in accordance with the mapping mode set by the mapping 
5 mode setting unit 103 shown in Fig. 1. 

The three mapping synthesis processing steps are 
described below. 

In step S506, the vertical- cylinder mapping synthesis 
is performed. The vertical -cylinder mapping generates a 

10 synthesized image mapped onto the vertical cylinder as shown 
in Fig. 3A. Fig. 6A schematically illustrates a synthesized 
image 601 obtained by the vertical cylinder mapping. The 

: horizontal coordinate is expressed by 0 (deg. ) and the 

vertical coordinate is expressed by v. Hence, the 

15 displayable range of the synthesized image 601 in the 

horizontal direction is from -180 deg. to +180 deg. 

Fig. 7 shows a sectional view of the vertical cylinder. 
Referring to Fig. 7, a sectional view 701 is taken along a 
line 610 of the synthesized image 601 obtained by the 

20 vertical-cylinder mapping as shown in Fig. 6A. The radius 

of the cylinder is one. A point P (6, v) in Fig. 6A 
indicates the same point as a point P in Fig. 7. In order 
to perform mapping synthesis, the point P (G, v) on the 
surface of the cylinder is transformed into an XYZ reference 

25 coordinate system as: 
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x = sin 8 
y = v 

z = cos 8 (2) 



The equation (1) are solved using the converted 
coordinates (x, y. z), thus obtaining the coordinates (x' # 
y' , z") corresponding to the point P (8, v) . Reference to 
the pixel levels of the coordinates is used to synthesize 

10 the image. Alternatively, as described above, it is 

possible to perform the transformation as u' = x'/z" and v' 
= y'/z', and to make reference to the pixel levels of the 
coordinates (u' , v' ) of the point P on the image. 

In step S506, the vertical- cylinder mapping synthesis 

15 processing obtains the synthesized image by generating pixel 

levels of coordinates corresponding to the point P (8, v) in 
Fig. 6A. 

In step S507, the horizontal-cylinder mapping synthesis 
processing generates the XYZ coordinates corresponding to a 
20 point P (u, on a synthesized image 602 obtained by the 

horizontal -cylinder mapping in Fig. 6B as follows: 



x = u 

y = sin (() 

z = cos <j) (3) 
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The horizontal -cylinder mapping synthesis processing 
obtains the pixel levels of the corresponding points by the 
same calculation as in the vertical -cylinder mapping 
synthesis processing. Hence, the description of the 
subsequent processing is omitted. 

In step S508, the spherical mapping synthesis 
processing obtains a synthesized image 603 by the spherical 
mapping, as shown in Fig. 6C. The horizontal axis of the 
synthesized image 603 indicates a horizontal viewing angle, 
and the vertical axis indicates a vertical viewing angle. 
The horizontal displayable range of the synthesized image 
603 is from -180 deg. to +180 deg.. and the vertical 
displayable range is from -90 deg. to +90 deg. As shown in 
Fig. 8, a point P (0, on the synthesized image 603 can be 
converted into the XYZ coordinates by: 

x = cos <j) sin 6 
y = sin <j) 

z = cos <|) cos 6 (4) 
The radius of the sphere is one . 

In a manner similar to the above synthesis processing, 
the synthesized image is generated by obtaining the pixel 
levels of the corresponding points. 
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A generated panoramic synthesized image is displayed on 
the display 122 by the synthesized image display unit 105 
shown in Fig. 1. The image can be stored by the synthesized 
image storage unit 107 in the recording medium 121, such as 
5 the magnetic disk. 

The recording medium 121 is not limited to the magnetic 
S disk. Needless to say, any known medium can be used as the 

recording medium 121. 
\ As described above, the mapping mode can be easily set 

lb by setting the appropriate mapping mode based on the image 
H placement information. By automatically setting the mapping 

m mode based on the placement information, the user is not 

O required to manually set the mapping mode when synthesizing 

an image. Hence, the panoramic synthesized image can be 
15 obtained without special knowledge. 

Alternatively, it is possible to notify the user of an 
appropriate mapping mode, and to make the user designate the 
actual mapping mode to be used when synthesizing an image. 
Although the user is required to perform some work in this 
20 case, the user is permitted to intentionally select a 
mapping mode differing from the appropriate one. This 
provides flexibility in selecting the mapping mode. 

Fig. 9 shows a panoramic image synthesis method and 
apparatus according to a second embodiment of the present 
25 invention. The characteristic portions of the second 
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embodiment include a mapping mode setting unit 110, an image 
synthesis processor 109, and a mapping mode changing unit 
108. The remaining portions are the same as the first 
embodiment and have the same reference numerals as those of 
5 the first embodiment. Hence, the description of the common 
portions is omitted. 

The basic operation of the mapping mode setting unit 
I 110 is the same as that of the first embodiment. Based on 

the placement information from a placement information 

10 setting unit 102, the mapping mode setting unit 110 sets the 

vertical-cylinder mapping mode when the images are captured 
in the one -dimensional horizontal direction, the horizontal - 
cylinder mapping mode when the images are captured in the 
one-dimensional vertical direction, and the spherical 

15 mapping mode when the images are captured in the two- 
dimensional tiled placement. The mapping mode setting unit 
110 of the second embodiment obtains focal length 
information of an image using a focal length obtaining unit 
(not shown) . Based on the focal length information, the 

20 mapping mode setting unit 110 generates a horizontal viewing 

angle Wh and a vertical viewing angle Wv of the image. 

When the vertical -cylinder mapping mode is set based on 
the placement information, the vertical viewing angle Wv is 
referred to and is compared with a predetermined reference 

25 value. When the vertical viewing angle Wv is larger than 
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the reference value, the mapping mode is changed from the 
vertical -cylinder mapping mode to the spherical mapping mode. 
Since the vertical -cylinder mapping mode performs vertical 
transformation in the same manner as the planar mapping mode, 
5 peripheral portions are noticeably deteriorated when the 

viewing angle is wide. As described above, the mapping mode 
can be selected more accurately by setting the mapping mode 
based on the focal length information as well. 
3= In the above example, even when the mapping mode is 

£0 selected based on the placement information, it is possible 

^ to refer to the horizontal viewing angle Wh and the vertical 

viewing angle Wv, and to change the mapping mode. 
Alternatively, the mapping mode can be selected from the 
beginning based on the placement information and the viewing 
15 angle information. 

The mapping mode changing unit 108 can be separated 
from the mapping mode setting unit 110, and the mapping mode 
can be changed in response to an instruction from the user. 
In this case, preferable mapping modes can be automatically 
20 selected based on the placement information and the viewing 
angle information, and the user can select one from among 
these mapping modes . 

With this arrangement, the preferable mapping mode can 
be selected even when the mapping mode is changed. 
25 The basic operation of the image synthesis processor 



- 22 - 



109 is the same as that of the first embodiment. Figs. 13A 
and 13B illustrate the vertical -cylinder mapping synthesis 
performed by the image synthesis processor 109 of the second 
embodiment. Referring to Fig. 13A, the shadowed portion 
5 indicates a margin indicating a region not included in 

;fS . visual fields of captured images. Basically, a synthesized 

*:* image is generated including the margin, as shown in Fig. 

E 5° 13A. Referring to Fig. 13B, a rectangular image region 900 

not including the margin can be computed, and the image 

Mo region 900 indicated by dotted lines can be displayed with 

O the synthesized image. 

rU Accordingly, a visually beautiful image can be 

O synthesized based on ill- looking images having Jagged 

peripheries . 

15 A synthesized image storage unit 107 can selectively 

store only the interior of the dotted region. Accordingly, 
it is possible to print or store a region without the margin. 

In a synthesized image, regions occupied by N-number of 
images can be generated and stored in angular representation. 

20 In other words, a region S k occupied by an image k is stored 

as (®min' ^min' ®max' 4>max) in a memor Y (not shown) Or the like. 

The mapping mode changing unit 108 arbitrarily changes 
the mapping mode for mapping a synthesized image. A basic 
mapping mode is automatically set by the mapping mode 
25 setting unit 110. The user is permitted to arbitrarily 
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change the mapping mode to a desired mapping mode after 
confirming the generated synthesized image. 

The operation of the mapping mode changing unit 108 is 
described. 

5 Fig. 14 illustrates a case in which the image 

synthesized by the vertical -cylinder mapping as shown in Fig. 
13A is transformed by the planar mapping mode. Referring 
: to Fig. 14, the larger the viewing angle, the more extended 

the image transformed by the planar mapping mode will be. 

10 Theoretically, the viewing angle ranges from -90 deg. to +90 
deg. The actual range is narrower than the theoretical 
range. Therefore, the mapping mode changing unit 108 

=- presets the range of the displayable viewing angle in 

accordance with the type of mapping mode to be changed. 

15 When changing the mapping mode, the mapping changing unit 

108 refers to the region S k occupied by each of the images 
generated by the image synthesis processor 109 among the 
entire image region. When the region S k exceeds the 
predetermined displayable region of the viewing angle set in 

20 accordance with the mapping type, a warning is issued, and 

only the synthesized image within the predetermined range of 
the viewing angle is generated and displayed. Accordingly, 
when the user inappropriately changes the mapping mode, the 
user is warned of the incorrect operation and obtains the 

25 synthesized image. 
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Fig. 15 schematically shows an image obtained by 
changing to the planar mapping mode by the mapping mode 
changing unit 108. Only the synthesized image within the 
range of the displayable viewing angle R 0 is generated. At 
5 the same time, the image region not including the margin is 

computed again and is indicated by the dotted lines . 

A synthesized image displaying unit 105 displays the 
-4 size of the synthesized image, which is used as the image 

,U information, not only in the pixel units, but also in the 

10 angle units. Fig. 16 shows the displayed image information. 
:;j The user sets the image range to be stored using an input 

^.j unit 111 in the pixel units or in the angle units. The set 

image range is indicated by dotted lines on the synthesized 
image displayed on the synthesized image displaying unit 105. 
15 In accordance with the set image range, the synthesized 

image storage unit 107 records and stores the synthesized 
image in a recording medium 121 or the like. It is also 
possible to convert the size of the image with an arbitrary 
scale when the synthesized image is stored. 
20 A panoramic image synthesis method and apparatus 

according to a third embodiment of the present invention 
restrict the number of mapping modes to be selected to two, 
i.e., the planar mapping mode and the vertical-cylinder 
mapping mode. The restriction on the types of mapping modes 
25 reduces the processing load. 
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Fig. 17 shows the panoramic image synthesis method, and 
apparatus of the third embodiment. Since the available 
types of mapping modes are limited to the planar mapping 
mode and the vertical -cylinder mapping mode in the third 
5 embodiment, a mapping mode setting unit 130 differs from 

===== those in the other embodiments. The operation of the 

fj= mapping mode setting unit 130 is described. 

The mapping mode setting unit 130 computes the number 
f.i of images in the horizontal direction and the vertical 

10 direction based on placement information set by a placement 

:H information setting unit 102. When there are m number of 

[~ images in the horizontal direction and n number of images in 

w the vertical direction, the values m and n are compared with 

preset reference values T m and T n of the numbers in the 
15 horizontal and vertical directions. Based on the comparison 
result, the mapping mode is set as below. When m ^ T m and n 
^ T n , the planar mapping mode is set. Otherwise, the 
vertical- cylinder mapping mode is set. Specifically, when 
the numbers of images are small, it is expected that the 
20 viewing angles of the image are not so large. In other 

words , it is expected that the peripheral portions of the 
image are not so distorted. Hence, when the numbers of 
images are equal to or less than the corresponding reference 
values, the planar mapping mode is selected. When the 
25 numbers of images exceed the corresponding reference values. 
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it is highly possible that the image becomes deteriorated 
due to the distortion. Hence, the vertical -cylinder mapping 
mode is selected . 

When the vertical-cylinder mapping mode is selected, it 
5 performs the coordinate transformation in the vertical 

direction in the same manner as the planar mapping mode. 
I The larger the viewing angle of the synthesized image in the 

; vertical direction, the more the peripheral portion becomes 

deteriorated, failing to generate a satisfactory image. To 
10 avoid this , as described in the second embodiment , the 

reference values of the displayable viewing angle are set in 
advance, and the synthesized image within these reference 
values is generated. 

In the above example, only the placement information is 
15 used to set the mapping mode. Alternatively, it is possible 

to obtain focal length f using an obtaining unit (not shown) . 
When m «s T m , n s T n , and f > T f , the planar mapping mode is 
set. Otherwise, the vertical -cylinder mapping mode is set. 
Although a combination of the planar mapping mode and 
20 the vertical- cylinder mapping mode is used in this example, 

the combination can be formed of other mapping modes . For 
example, when m a T m and n <; T n , the vertical- cylinder 
mapping mode is set. When m <; T m and n > T n , the horizontal- 
cylinder mapping mode is set. Also, it is possible to add 
25 the planar mapping mode to the mapping modes used in the 
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first and second embodiments, and the planar mapping mode 
can be included in the above combination. Accordingly, the 
mapping mode can be set by simple computation processing 
using the horizontal and vertical numbers of images as the 
5 placement information. 

A panoramic image synthesis method according to a 
fourth embodiment of the present invention includes a step 
of computing the relationship between coordinate spaces of 
input images irrespective of the mapping mode, and 

10 separately, a transforming step using coordinate 

- transformation parameters which differ in accordance with 

the mapping mode. When synthesizing an image, the two 

I separate steps are combined so as to synthesize the image in 

which the mapping mode is changeable. Hence, the mapping 

15 mode can be changed by changing over the coordinate 

transformation parameters stored in association with the 
mapping mode, thereby easily obtaining synthesized images in 
various mapping modes . 

Referring to Fig. 19, images to be used by the 

20 panoramic image synthesis method and apparatus of the fourth 

embodiment are captured by an image capturing person who 
inputs an image in an image region 11 at a state 11 while 
rotating at an arbitrary position, an image in an image 
region 12 at a state 12, and an image in an image region 13 

25 at a state 13. 
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The rotating direction includes the Y-axis rotation 
(hereinafter referred to as "panning") or the X-axis 
rotation (hereinafter referred to as "tilting"), as shown in 
Fig. 19. Referring to Fig. 19, the images are input and 
5 combined into a panoramic image which covers an arbitrary 
viewing angle. 

];? The relationship between the panoramic image and 

mapping is described. Fig. 20 is a sectional view of the 
illustration shown in Fig. 19 at the states 11 to 13 when Y 

ib is equal to zero. The origin O and the coordinate axes X, Y, 

and Z are set based on the state 11. Focal length is 
expressed by f , and values 4> 12 . and § 23 , express panning 

□ angles. There are sensor surfaces (image planes) 1301, 1302, 

and 1303. 

15 In order to combine a plurality of images captured in 

different directions into a single image, it is necessary to 
map the images onto a single surface. This can be done by 
the cylinder mapping mode of projecting the images disposed 
on the sensor surfaces onto a cylindrical surface 1310 or by 

20 the planar mapping mode of projecting the images onto a 

plane 1320, as shown in Fig. 20. 

Referring to Figs. 21A to 21C, the mapping modes are 
described. 

Fig. 21A illustrates the planar mapping mode of 
25 projecting the images disposed on the sensor surfaces onto a 
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plane 1401 around the origin O. 

With the planar mapping mode, it is possible to 
generate a synthesized image which is similar to an image 
captured by a wide-angle lens. A straight -line portion of a 
5 subject is represented as a straight line in the synthesized 

image. The range of the displayable viewing angle is 
limited to below 180°. Even when the viewing angle is below 
180°, the larger the viewing angle, the more the peripheral 
portion is distorted and deteriorated. Therefore, it is 

10 appropriate to set the viewing angle of the synthesized 
image to approximately 120°. 

Fig. 21B illustrates the cylinder mapping mode. The 
images on the sensor surfaces are projected onto a cylinder 
1402 with the center line passing through the origin O. 

15 The horizontal direction of the image obtained by 

expanding the lateral side of the cylinder is expressed by 
an angle which can express the entire perimeter of 360°. 
In contrast, the vertical direction is the same as that in 
the planar mapping mode, and the displayable horizontal 

20 viewing angle is below 180°. As in the planar mapping mode, 

the larger the viewing angle, the more the peripheral 
portion is distorted and deteriorated. In the cylinder 
mapping mode, a straight-line portion of a subject is 
distorted in accordance with the position of the image. 

25 In the cylinder mapping mode, two types of cylinders 
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are used. One has the X-axis as the rotation axis, and the 
other has the Y-axis as the rotation axis. It is thus 
effective to select the rotation axis in accordance with the 
image capturing condition. When an image is captured by 
5 panning, a cylinder (vertical cylinder) rotating around the 

Y-axis is used. When an image is captured by tilting, a 
cylinder (horizontal cylinder) rotating around the X-axis is 
used. 

Fig. 21C shows the spherical mapping mode of projecting 
10 images disposed on the sensor surfaces onto a spherical 

surface 1403 around the origin O. 

In an image obtained by the spherical mapping mode, 
; both the horizontal direction and the vertical direction are 

expressed by angles. Hence, it is possible to generate an 
15 all-around panoramic image. In the spherical mapping mode, 

a straight-line portion of a subject is distorted in 
accordance with the position of the image. 

The spherical mapping mode is suitable for representing 
a synthesized image generated by combining panned images and 
20 tilted images . 

Accordingly, there are various types of mapping modes for 
synthesizing an image. It is thus effective to select the 
mapping mode in accordance with the type of subject and the 
image capturing condition. 
25 Fig. 18 shows the panoramic image synthesis method and 
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apparatus according to the fourth embodiment of the present 
invention. 

Referring to Fig. 18, an image input unit 1701 inputs 
images to be used for image synthesis . The images can be 
5 input by reading images recorded in a recording medium (not 

shown) . Alternatively, the images can be input from a 
memory (not shown). Video signals obtained from an imager, 
such as a digital camera, can be digitized and input as the 
input images. 

10 A memory 1702 stores various types of information, such 

as image information and parameters used for synthesis 
processing. 

A microprocessing unit (MPU) 1703 controls the entire 
synthesis processing and performs various arithmetic 

15 processing. 

A synthesis parameter generator 1704 generates 
coordinate- space transformation parameters indicating the 
positional relationship between adjacent images . The 
synthesis parameter generator 1704 also generates various 

20 synthesis parameters required for generating a synthesized 

image by computing gray- scale correction parameters and an 
adjoining position between adjacent images. Although the 
gray- scale correction parameters are computed in the 
embodiment, so-called color-matching parameters for matching 

25 brightness and chrominance between adjacent images may be 
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computed. The synthesis parameter generator 1704 is 
connected to the memory 1702 and the MPU 1703. Information 
including image data and various synthesis parameters is 
input to and output from the memory 1702 . Control 
5 information is input to and output from the MPU 1703. 

An image synthesis converter 1705 combines a plurality 
=;-" = of images using the coordinate transformation parameters in 

l ~ accordance with the given mapping mode and the synthesis 

ft parameters including the coordinate -space transformation 

lb parameters generated by the synthesis parameter generator 

1704. Specifically, the image synthesis converter 1705 
^ gains access to the memory 1702 based on the control 

Q information from the MPU 1703 and reads the image data and 

the synthesis parameters including the coordinate- space 
15 transformation parameters. The image synthesis converter 

1705 then generates a synthesized image by performing 
coordinate transformation processing or the like. The 
obtained synthesized image is written to the memory 1702. 
It is necessary to distinguish between the coordinate 
20 transformation parameters and the coordinate -space 

transformation parameters . 

The coordinate transformation parameters are parameters 
set for each mapping mode. 

In contrast, the coordinate -space transformation 
25 parameters are parameters indicating the relationship 
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between a coordinate plane of one image and a coordinate 
plane of another image among a plurality of images . 

A recording medium 1706 includes a floppy disk or the 
like. The recording medium 1706 records a synthesized 
5 panoramic image . 

An operating unit 1707 is used to set focal length and 
to select the mapping mode. The specific operation can be 
performed with a GUI using a mouse or the like. 
Alternatively, commands can be directly input using a 
10 keyboard. 

A display unit 1708 displays a panoramic synthesized 
image, a mapped/ transformed image, GUI, and the like. 

The operation of the panoramic image synthesis method 
and apparatus according to the fourth embodiment is 
15 described. 

The image input unit 1701 inputs images in which 
adjacent images partially overlap with each other. The 
memory 1702 stores the input images. Images to be input can 
be selected from candidate images displayed on the display 
20 unit 1708 based on an instruction from the operating unit 

1707. Alternatively, a single set of panoramic images can 
be automatically selected and input based on the attribute 
of an image file. 

The operating unit 1707 sets image capturing focal 
25 length f of the images to be used for synthesis processing. 
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The focal length f can be set by inputting a value of the 
focal length or by selecting a value from among a plurality 
of candidate values. When the image capturing focal length 
is preset as the attribute of an image stored in the memory, 
5 that value can be used as the focal length f . 

When the image information and the image capturing 
focal length are set, the synthesis parameter generator 1704 
generates the parameters used for combining the adjacent 
images . 

10 Fig. 22 is a flowchart showing a process of generating 

the synthesis parameters. 

In step S1501, two images to be used for setting the 

synthesis parameters are set. In step S1502, corresponding 

points in the overlapping region of the two set images are 
15 extracted. The corresponding points can be extracted by a 

conventional technique, such as the template matching method 

or the correlation method. 

In order to enhance processing efficiency, it is 

advantageous to employ a conventional coarse -to -fine 
20 hierarchical method of extracting the corresponding points. 

In step S1502, the coordinate- space transformation 

parameters between the images are computed based on 

information about the extracted corresponding points. 

Referring to Fig. 23, an example of computing the 
25 coordinate -space transformation parameters is described. 
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Fig. 23 shows a sectional view taken through the XZ 
plane of the panned images shown in Fig. 19 in which Y is 
equal to zero. Sensor surfaces 1601 and 1602 correspond to 
the adjacent captured images. When the origin of the XYZ 
coordinate system corresponding to the sensor surface 1601 
is expressed by O, and the origin of the X'Y'Z' coordinate 
system corresponding to the sensor surface 1602 is expressed 
by O', the coordinate -space transformation parameters 
indicate the relationship between the XYZ coordinate system 
and the X'Y'Z' coordinate system. When the corresponding 
points obtained by the corresponding point extraction 
processing in step S1501 are expressed by a point P and a 
point P' , the relationship between the two points P and P" 
can be concluded as follows using a rotation parameter R and 
a translation parameter T shown in Fig. 23: 

P ' = R • P + T (5) 

In the equation (5), R represents a rotation matrix 
obtained from rotational components (0, c|>, tp) rotating 
around the axes, respectively. The rotation matrix R can be 
expressed as follows : 

/m x m 2 m 3 \ 
R = m 4 m 5 m 6 
\m 7 m 8 m 9 y 
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wherein 






m i : 


cos (<())cos (<()) 






m 2 : 


— sin (([>)cos(0) 


+ cos((()) sin((j)) sin( 




m 3 : 


sin(()))sin(6) 


+ cos((j)) sin((())cos(0 
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m 4 : 


sin((|))cos ((()) 






m 5 : 


COS((())COS(0) 


+ sin((()) sin((j)) sin(0 




m 6 : 


-cos((())sin(6) 


+ sin((f))sin((t))cos( 




m 7 : 


-sin( (f>) 






m 8 : 


cos(({>)sin(0) 




ID 


m 9 : 


cos((j))cos(6) 





A synthesized matrix M of the rotation matrix R and the 
translation matrix T is expressed as follows: 




(7) 

15 

The synthesized matrix M can be computed using a 
nonlinear least -squares method or the like based on the 
information about the corresponding points obtained by the 
corresponding point extraction processing. It is also 
20 possible to use other conventional techniques to compute the 
parameters . 

Accordingly, the computation of the relationship 
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between the coordinate system XYZ and the coordinate system 
X'Y'Z' provides a clue to identify, for example, a point on 
the coordinate system X'Y'Z' that corresponds to a point P 
(x, y, z). 

5 A specific example of this is given hereinafter. The 

^ synthesized matrix M is computed as the coordinate- space 

transformation parameters because the synthesized matrix M 
:j represents the corresponding relationship between the points 

on the images . In other words , the synthesized matrix M 
10 represents the positional relationship in space when the 

~~i images are captured based on the arbitrarily input two- 

f U dimensional image information. 

I;3 Therefore, it is only necessary to map the images onto 

an appropriate common surface, thus generating an 

15 appropriate synthesized image. 

In step S1504, the process generates correction 
parameters for making the adjoining part between the 
adjacent images less noticeable by matching the gray scale 
of the adjacent images. 

20 In step S1505, the position of the adjoining part 

between the adjacent images is set using the generated 
coordinate transformation parameters or the gray-scale 
correction parameters. The position of the adjoining part 
is set using a difference or a correlation value between 

25 pixels re-sampled based on the coordinate-space 
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transformation parameters. 

In step S1506, it is determined whether the generation 
of the parameters between all pairs of adjacent images is 
completed. If the determination is negative, two images to 
5 be combined are again set in step S1501, and the synthesis 
processing is continued. 
^ If the determination is affirmative, a reference 

l ]z position to be used when combining all images is set in step 

W S1507. 

it) In step S1508, the already- obtained coordinate -space 

O transformation parameters are transformed based on the set 

f!j reference position. 

O Accordingly, the synthesis parameters are generated by 

the synthesis parameter generator 1704 shown in Fig. 18. 

15 The synthesis parameters are stored in the memory 1702. 

When the synthesis parameters are generated, a 
panoramic synthesized image is generated in a mapping mode 
selected by the image synthesis converter 1705. Referring 
to Figs. 24A and 24B, processing performed by the image 

20 synthesis converter 1705 is described. 

Figs. 24A and 24B illustrate a case in which a 
synthesized image is generated by the cylinder mapping 
transformation. Fig. 24A shows a panoramic synthesized 
image 1710 obtained by the cylinder mapping. The processing 

25 performed by the image synthesis converter 1705 is described 
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using an example of generating a pixel I k on the synthesized 
image 1710. Fig. 24B schematically shows the relationship 
between sensor surfaces 1712 and 1713 at image capturing 
positions. A cylindrical surface 1711 shows a sectional 
5 view taken through Y = Y k of the synthesized image 1710. In 

*~ order to make the description simple, the above translation 

component T is omitted in Fig. 24B. 

Referring to Fig. 24B, the coordinates of the point I k 
if:' (c^, Y k ) on the cylindrical surface 1711 are transformed into 

ib a point P (x, y, z) on the XYZ coordinate system. This 
coordinate transformation is expressed as follows: 



f • sin((|) k ) 

f • costy (8) 



When the point P is represented as a point P" (x', y* , 
z') on the X'Y'Z" coordinate system, the coordinate 
transformation between the point P and the point P' is 
20 performed using a coordinate- space transformation parameter 

M of the equations (8) generated by the synthesis parameter 
generator 1704. The coordinate -space transformation 
parameter M is expressed as follows: 



25 
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fx\ 


Y' 




y 




= M 




z' 




z 


a; 




A, 



(9) 

Next, coordinates of a point P" on the sensor surface 
Ff5 1713 corresponding to the point P' are obtained by the 

|ij following equations : 

f • * 

z' 

f 

z' 

f 

(10) 

Reference to a pixel at the coordinates (x", y" ) on the 
10 sensor surface 1713 (z" = f) is made using the equations 

(10), thus obtaining the value of the pixel I k on the 
synthesized image 1710. 

The above processing is performed for each pixel on the 
synthesized image 1710, thus generating the synthesized 
15 image which is cylinder -mapped and transformed. 

The synthesized image is stored in the memory 1702 and 
is displayed on the display unit 1708. As circumstances 
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demand, the image is recorded in the recording medium 1706. 

Changing processing for changing the mapping mode is 
described. 

The user employs the operating unit 1707 to select an 
5 arbitrary mapping mode and to instruct the mapping mode to 

be changed. The MPU 1703 sets coordinate transformation 

processing in accordance with the selected mapping mode. 
'_== The image synthesis converter 1705 performs the image 

^ conversion processing, as described above. 

10 When the mapping mode is changed, the coordinate 

transformation processing using the equations (8) is 
S H replaced with new one corresponding to the newly selected 

mapping mode. 

For example, when the mapping mode is changed to the 
15 planar mapping mode, each pixel on a synthesized image 

corresponds to the coordinates on a sensor surface. Hence, 
the coordinate transformation equations (8) corresponding to 
the pixel I k (X k , Y k ) to be generated are as follows: 



(11) 



When the mapping mode is changed to the spherical 
25 mapping mode, sampling is performed so that both the 
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horizontal direction and the vertical direction provide 
isogonal pitches. The coordinate transformation 
corresponding to the pixel I k ((f^, 0 k ) is performed by the 
following equations : 

x = f • cos(0 k ) sinCcj^) 
y = f • cos(6 k ) 

z = f • cos(0 k ) cosl^) (12) 

The coordinate- space transformation parameter M 
representing the relationship between the images is the same 
for all mapping modes . The coordinate transformation 
equations are changed among the equations (8), (11), and 
(12) in accordance with the type of selected mapping mode. 
Accordingly, it is possible to generate a synthesized image 
in an arbitrary mapping mode. 

Whenever the mapping mode is changed, a conventional 
image synthesis method substitutes the coordinates of the 
corresponding points for the equations based on the mapping 
mode, and generates the synthesis parameters. Based on the 
synthesis parameters, the coordinates of all pixels are 
transformed. 

In contrast, the panoramic image synthesis method of 
the fourth embodiment performs the synthesis processing by 
first obtaining the corresponding relationship between the 
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points on the images as the coordinate -space transformation 
parameters, and subsequently changing the coordinate 
transformation processing in accordance with the mapping 
mode. Therefore, when the mapping mode is changed, it is 
5 only necessary to map and transform the input images using 
the already- obtained coordinate- space transformation 
parameters and the coordinate transformation processing. In 
other words, it is not necessary to again compute the 
: parameters. Hence, the panoramic image synthesis method of 

10 the fourth embodiment can generate synthesized images in 

various mapping modes in an extremely efficient manner. In 
particular, the method can substantially reduce the 
- processing time when the mapping mode is changed. At the 

same time, the method can generate a synthesized image of 
15 excellent quality because the method does not transform a 

synthesized image. 

A panoramic image synthesis method and apparatus 
according to a fifth embodiment of the present invention 
have the following characteristics: 
20 1) A user can arbitrarily change a reference position, 

and a synthesized image can be converted based on the 
changed reference position. 

2) An adjoining part can be corrected. 

3) A synthesized image can be restored to a previous 
25 synthesis result. 
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The operation of the panoramic image synthesis method 
and apparatus of the fifth embodiment is described. The 
overall structure of the fifth embodiment is the same as 
that shown in Fig. 18. The panoramic image synthesis method 
5 and apparatus of the fifth embodiment operate in a manner 

similar to those of the first embodiment up to the point at 
which a synthesized image is generated in a predetermined 
mapping mode and is displayed on a display unit . Hence , the 
description of the common parts is omitted. 

10 Referring to Figs. 25A, 25B, 26A, and 26B, processing 

for changing a reference position according to the fifth 
embodiment is described. Fig. 25A shows a case in which a 
: synthesis result is displayed by planar mapping. A 

synthesized image 1801 shown in Fig. 25A is obtained by 

15 combining images provided by capturing a building by 

vertical tilting. The synthesized image 1801 is reduced in 
size at the top and has a trapezoidal shape. The 
synthesized image 1801 has a reference position 1802. 
Fig. 25B shows the relationship between a subject 

20 building 1803 and sensor surfaces 1804 and 1806 at different 
states in which the subject building 1803 is captured by 
tilting. A projection surface 1807 displays a synthesized 
image with a reference position 1808 of the synthesized 
image . 

25 In order to change the reference position, the 
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reference position 1802 shown in Fig. 25A is moved by the 
operating unit 1707 shown in Fig. 18, such as a mouse or the 
like. Alternatively, the reference position can be moved by 
an instruction from a keyboard, which designates the 
5 coordinates of a destination, or by an arrow key or the like. 

The changed position is sent to an MPU 1703. The MPU 
1703 outputs movement information to an image synthesis 
converter 1705. The image synthesis converter 1705 
generates a correction matrix M' for correcting the 
10 synthesized image based on the movement information. 

The image synthesis converter 1705 corrects the 
coordinate- space transformation parameters M of all the 
images using the correction matrix M' , thus generating a 
synthesis conversion matrix M2 : 

15 

M2 = M • M' (13) 

Subsequently, the synthesized image is generated in a 
selected mapping mode as in the first embodiment. The 
20 synthesized image is stored in a memory 1702 and is 

displayed on a display unit 1708. 

Fig. 26A shows a result of moving the reference 
position and converting the synthesized image. A resultant 
synthesized image 1901 has a current reference position 1902. 
25 Fig. 26B shows the relationship between the subject and 
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the sensor surfaces. 

The reference position is moved to a reference position 

1908. A projection surface 1907 on which the synthesized 

image is projected is set based on the reference position 
5 1908. Using the projection surface 1907, the synthesized 

image is generated . 

Since the projection surface 1907 is parallel to a 

lateral side of the subject, an undistorted synthesized 

image 1901 can be generated. 
10 There are conventional methods and apparatuses for 

arbitrarily setting a reference plane corresponding to the 

projection surfaces 1807 and 1907 and forming a synthesized 
~- image on the reference plane. However, these conventional 

methods and apparatuses are not advantageous since they must 
15 re-compute synthesis parameters from the beginning whenever 

the reference plane is changed. 

The panoramic image synthesis method and apparatus of 

the fifth embodiment are required to only apply the 

coordinate space changing parameters corresponding to the 
20 reference position to the already-computed coordinate -space 

transformation parameters and to perform computation. 

Accordingly, usability is significantly increased. The user 

is capable of setting the reference position at a desired 

position when generating the synthesized image. This 
25 flexibly complies with geometrical correction, such as 
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tilting and shifting correction. 

Next, correction processing of an adjoining part is 
described. Fig. 27A shows images 1101 and 1102 captured so 
as to partially overlap with each other. Fig. 27B shows a 
5 synthesized image 1103 of the images 1101 and 1102. An 

adjoining part of the synthesized image 1103 is indicated by 
dotted lines . 

The synthesized image 1103 is out of alignment at 
Ji portions near the adjoining part. When a synthesized image 

10 is not aligned, the panoramic image synthesis method and 

□ apparatus of the fifth embodiment correct the alignment , as 

-4 shown in Figs. 28A and 28B. 

O Referring to Fig. 28A, the two captured images 1101 and 

1102 are moved by the operating unit 1707 shown in Fig. 18, 

15 and the overlapping portions of the two images 1101 and 1102 

are aligned so as to substantially correspond with each 
other. The alignment information is sent to the MPU 1703. 
The MPU 1703 outputs the alignment information to a 
synthesis parameter generator 1704, and the synthesis 

20 parameter generator 1704 uses the information for extracting 

the corresponding points. 

The synthesis parameter generator 1704 can extract the 
corresponding points in a state in which the overlapping 
portions are aligned to a certain degree. Therefore, 

25 extraction accuracy and reliability are improved. 
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Referring to Fig. 28B, points on the subject on the 
images 1101 and 1102 are directly set as the corresponding 
points by the operating unit 1707 shown in Fig. 18. A pair 
of points 1112A and 1112B, and a pair of points 1113A and 
5 1113B are set as pairs of corresponding points. 

The MPU 1703 obtains information about the pairs of 
corresponding points, and sends the information to the 
== synthesis parameter generator 1704. 

-I The synthesis parameter generator 1704 uses the 

10 information about the pairs of corresponding points to 

estimate the coordinate -space transformation parameters. 
Since the obtained pairs of corresponding points are aligned 
by manual operation, it is possible to increase accuracy in 
the pairs of corresponding points by performing the template 

15 matching method or the correlation method in the vicinity of 

the pairs of corresponding points , and to use the 
information about the more accurate corresponding points to 
estimate the coordinate- space transformation parameters. 
Fig. 28C shows a corrected synthesized image 1114. 

20 When compared with the image 1103, the synthesized image 

1114 has a smooth adjoining part. This correction 
processing can be repeated until the user is satisfied with 
the result . 

When the resultant image obtained by changing the 
25 mapping mode and correcting the adjoining part is 
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satisfactory, there is no problem. However, there are cases 
in which the previous image is more satisfactory than the 
resultant image. In order to solve this problem, the 
panoramic image synthesis method and apparatus of the fifth 
5 embodiment store the synthesis parameters and the image 

information about the synthesis results in a memory 1702 
whenever a synthesized image is generated until a series of 
image synthesis processing steps is completed. 

For example, the synthesis result 1103 shown in Fig. 
10 27B is set as a first synthesis result in which synthesis 

counters are set such that Csl = 1 and Cs2 =1. At the same 
time, the various synthesis parameters and the synthesized 
image are stored at the memory position corresponding to the 
synthesis counters Csl = 1 and Cs2 = 1. The synthesis 
15 counter Csl is a counter incremented by one whenever 

changing/correction processing is performed. The synthesis 
counter Cs2 is a counter incremented by one whenever the 
current state returns a one -step previous state and 
changing/correction processing is again performed in that 
20 previous state. 

When the synthesized image 1114 is obtained by the 
changing/correction processing, the synthesis counters are 
set such that Csl = 2 and Cs2 = 1, and the synthesis 
parameters and the synthesized image are stored at the 
25 memory position corresponding to the synthesis counters Csl 
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= 2 and Cs2 = 1. 

When a synthesized image 1120 shown in Fig. 29 is 
obtained by the changing/correction processing, the 
synthesis counters are set such that Csl = 3 and Cs2 = 1, 
and the synthesis parameters and the synthesized image are 
stored at the memory position corresponding to the synthesis 
counters Csl = 3 and Cs2 = 1. 

When the user gives an "undo" instruction by the 
operating unit 1707, the synthesis counter Csl is decreased 
by one to be Csl = 2. The synthesis parameters and the 
synthesis resultant image corresponding to the synthesis 
counters Csl = 2 and Cs2 = 1 are read from the memory 1702. 
The synthesized image is displayed on the display unit 1708. 
The read parameters are replaced with the current parameters. 

At this point, a synthesized image 1131 shown in Fig. 
30 is displayed. When changing/correction processing is 
performed, the Cs counters are incremented by one, thus 
setting Csl = 2 and Cs2 = 2. The synthesis parameters and 
the synthesized image are stored at the memory position 
corresponding to the synthesis counters Csl = 2 and Cs2 = 2. 
In this state, a synthesized image 1133 shown in Fig. 30 is 
provided . 

When further correction is performed, a synthesized 
image 1134 is displayed, and the synthesis counters become 
Csl = 2 and Cs2 = 3. 
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The above processing is performed by the MPU 1703 and 
the memory 1702. Since all images generated whenever 
changing/correction processing is performed are stored in 
the memory 1702, the user can easily restore the image to a 
5 previously-generated arbitrary image. 

The operating unit 1707 indicates the "undo" directions 
using a left arrow and a right arrow. 

Alternatively, any conventional technique can be used 
to designate the direction. 
10 For example, the values of the counters Csl and Cs2 can 

be input, thereby restoring the image to a previously- 
generated arbitrary image. 
H In the above example, the synthesis parameters and the 

synthesized images are stored. When the synthesized images 
15 are large in size or the number of the synthesized images is 

large, these data may impose a burden on the memory capacity. 
In such a case, it is effective to store only the synthesis 
parameters and to sequentially generate an image using the 
synthesis parameters. 
20 According to the panoramic image synthesis method and 

apparatus of the fifth embodiment, correction of a 
synthesized image, such as tilting and shifting correction, 
is performed in a flexible and efficient manner. 

The above embodiments can be applied to a system 
25 including a plurality of apparatuses, such as a host 
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computer, an interface device, a reader, and a printer. 
Alternatively, these embodiments can be applied to an 
apparatus including a single device, such as a copying 
machine or a facsimile machine. 

The present invention covers the following case. In 
order to perform the functions of the above embodiments, a 
computer in an apparatus or a system connected to various 
devices so as to operate the devices is provided with 
software program codes for executing the functions. The 
computer (CPU or MPU) in the system or the apparatus 
operates the various devices in accordance with the stored 
program. 

In this case, the software program codes execute the 
functions of the above embodiments. The present invention 
covers the program codes and a medium for supplying the 
computer with the program codes , such as a storage medium 
for storing the program code. 

The storage medium for storing the program code 
includes a floppy disk, a hard disk, an optical disk, a 
magneto-optical disk, a CD-ROM, a magnetic tape, a non- 
volatile memory card, or a ROM. 

The present invention is not limited to a case in which 
the functions of the embodiments are performed by executing 
the supplied program codes by the computer. It also covers 
a case in which the program codes perform the above 
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functions in cooperation with an operating system (OS) 
running in the computer or with other application software. 

The present invention also covers the following case. 
After the program codes are stored in a memory of an add-on 
board of the computer or an add-on unit connected to the 
computer, a CPU of the add-on board or the add-on unit 
performs part or entirety of the actual processing based on 
instructions from the program codes, thereby performing the 
functions of the above embodiments. 

While the present invention has been described with 
reference to what are presently considered to be the 
preferred embodiments, it is to be understood that the 
invention is not limited to the disclosed embodiments. On 
the contrary, the invention is intended to cover various 
modifications and equivalent arrangements included within 
the spirit and scope of the appended claims. The scope of 
the following claims is to be accorded the broadest 
interpretation so as to encompass all such modifications and 
equivalent structures and functions. 
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WHAT IS CLAIMED IS: 

1. An image synthesis method comprising: 

a placement information obtaining step of obtaining 
placement information about a plurality of images in which 
adjacent images have a common subject region; and 

a synthesis step of combining the images using a 
mapping mode in accordance with the placement information. 

2. An image synthesis method according to Claim 1, 
further comprising: 

a focal length obtaining step of obtaining focal length 
information of each of the images; 

wherein the mapping mode in accordance with the focal 
length information and the placement information is used in 
said synthesis step. 

3 . An image synthesis method according to Claim 1 , 
further comprising a changing step of changing the mapping 
mode. 

4. An image synthesis method according to Claim 3, 
further comprising a generating step of issuing, when an 
image formed by changing the mapping mode in said changing 
step does not comply with a predetermined condition set in 
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accordance with the mapping mode, a warning and generating a 
synthesized image in accordance with the predetermined 
condition. 

5 . An image synthesis method according to Claim 1 , 
further comprising a displaying step of displaying a 
cuttable rectangular region without a margin in the 
synthesized image. 

6. An image synthesis apparatus comprising: 
placement information obtaining means for obtaining 

placement information about a plurality of images in which 
adjacent images have a common subject region; and 

synthesis means for combining the images using a 
mapping mode in accordance with the placement information. 

7 . An image synthesis apparatus according to Claim 6 , 
further comprising: 

focal length obtaining means for obtaining focal length 
information of each of the images; 

wherein said synthesis means uses the mapping mode in 
accordance with the focal length information and the 
placement information. 

8. An image synthesis apparatus according to Claim 6, 
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further comprising changing means for changing the mapping 
mode . 

9. An image synthesis apparatus according to Claim 8, 
further comprising generating means for issuing, when an 
image formed by changing the mapping mode by said changing 
means does not comply with a predetermined condition set in 
accordance with the mapping mode, a warning and generating a 
synthesized image in accordance with the predetermined 
condition. 

10. An image synthesis apparatus according to Claim 6, 
further comprising display means for displaying a cuttable 
rectangular region without a margin in the synthesized image. 

11. A computer-readable storage medium having recorded 
thereon a program for implementing a computer -implement able 
image synthesis method for combining a plurality of images, 
said program comprising: 

a placement information obtaining step of obtaining 
placement information about a plurality of images in which 
adjacent images have a common subject region; and 

a synthesis step of combining the images using a 
mapping mode in accordance with the placement information. 
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12. A computer- readable storage medium according to 
Claim 11, said program further comprising: 

a focal length obtaining step of obtaining focal length 
information of each of the images; 

wherein the mapping mode in accordance with the focal 
length information and the placement information is used in 
said synthesis step. 

13. A computer -readable storage medium according to 
Claim 11, said program further comprising a changing step of 
changing the mapping mode. 

14. An image synthesis method conforming to a 
plurality of mapping transformation modes, comprising: 

a generating step of generating, prior to performing 
mapping transformation of input images, coordinate- space 
transformation parameters for transforming a coordinate 
space of one image among the images into a coordinate space 
of another image; and 

an image synthesis step of combining the images based 
on a given mapping mode and the coordinate- space 
transformation parameters. 

15. An image synthesis method according to Claim 14, 
further comprising: 
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a changing instruction step of issuing a mapping mode 
changing instruction; 

wherein, in said image synthesis step, coordinate 
transformation parameters set for each of the mapping modes 
are changed and the input images are again combined in 
accordance with the changing instruction issued in said 
changing instruction step. 

16. An image synthesis method according to Claim 14, 
further comprising: 

a reference position setting step of arbitrarily 
setting a reference position of a synthesized image; 

wherein, in said image synthesis step, the images are 
combined in accordance with the reference position set in 
said reference position setting step. 

17. An image synthesis method according to Claim 14, 
further comprising a storage step of storing a generated 
panoramic synthesized image, and coordinate transformation 
parameters and the coordinate- space transformation 
parameters, which are used for generating the panoramic 
synthesized image, every time the synthesized image is 
generated in said image synthesis step. 

18. An image synthesis apparatus conforming to a 
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plurality of mapping transformation modes, comprising: 

generating means for generating, prior to performing 

mapping transformation of input images, coordinate -space 

transformation parameters for transforming a coordinate 

space of one image among the images into a coordinate space 

of another image; and 

image synthesis means for combining the images based on 

a given mapping mode and the coordinate- space transformation 

parameters . 

19. An image synthesis apparatus according to Claim 18 
further comprising: 

changing means for changing the mapping mode; 

wherein said image synthesis means changes coordinate 
transformation parameters set for each of the mapping modes 
and again combines the input images in accordance with a 
mapping mode changing instruction. 

20. An image synthesis apparatus according to Claim 18 
further comprising: 

reference position setting means for arbitrarily 
setting a reference position of a synthesized image; 

wherein said image synthesis means combines the images 
in accordance with the reference position set by said 
reference position setting means. 
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21. An image synthesis apparatus according to Claim 18, 
further comprising storage means for storing a generated 
panoramic synthesized image, and coordinate transformation 
parameters and the coordinate -space transformation 
parameters, which are used for generating the panoramic 
synthesized image, every time the synthesized image is 
generated by said image synthesis means . 

22. A computer- readable storage medium having recorded 
thereon a program for implementing a computer -implement able 
panoramic image synthesis method conforming to a plurality 
of mapping transformation modes, said program comprising: 

a generating step of generating, prior to performing 
mapping transformation of input images, coordinate- space 
transformation parameters for transforming a coordinate 
space of one image among the images into a coordinate space 
of another image; and 

an image synthesis step of combining the images based 
on a given mapping mode and the coordinate- space 
transformation parameters. 

23. An image synthesis method for combining input 
images into a synthesized image, comprising: 

an editing step of editing the synthesized image; and 
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a storage step of storing the synthesized image 
obtained every time the synthesized image is edited. 

24. An image synthesis method according to Claim 23, 
wherein a mapping mode is changed and the images are 
combined in said editing step. 

25. An image synthesis method according to Claim 23, 
wherein an adjoining part of the images is corrected in said 
editing step. 
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ABSTRACT OF THE D TKCT.OSURE 



An image synthesis method includes a placement 
information obtaining step of obtaining placement 
information about a plurality of images in which adjacent 
images have a common subject region, and a synthesis step of 
combining the images using a mapping mode in accordance with 
the placement information. Prior to performing mapping 
transformation of the input images, coordinate- space 
transformation parameters for transforming the coordinate 
space of one image among the images into the coordinate 
space of another image are generated. The images are 
combined based on a given mapping mode and the coordinate - 
space transformation parameters . 
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